Surface coupling in Bi2Se3 ultrathin films by screened Coulomb interaction

Single-particle band theory has been very successful in describing the band structure of topological insulators. However, with decreasing thickness of topological insulator thin films, single-particle band theory is insufficient to explain their band structures and transport properties due to the existence of top and bottom surface-state coupling. Here, we reconstruct this coupling with an equivalently screened Coulomb interaction in Bi2Se3 ultrathin films. The thickness-dependent position of the Dirac point and the magnitude of the mass gap are discussed in terms of the Hartree approximation and the self-consistent gap equation. We find that for thicknesses below 6 quintuple layers, the magnitude of the mass gap is in good agreement with the experimental results. Our work provides a more accurate means of describing and predicting the behaviour of quasi-particles in ultrathin topological insulator films and stacked topological systems.

In several previous works, the effects on the mass gap in the single-particle spectrum of such systems has been studied as a function of thickness of samples, i.e as a function of quintuplet layers (QL). It has been found that this mass gap can change sign from positive (topologically trivial) to negative (topologically non-trivial) as thickness is varied. The authors of the present paper point out that i) the gap reclosing (necessary for a change of sign in m) has not been observed, ii) the experimentally reported mass gaps are far larger than predicted, and iii) the absence of ballistic 1D edge states means that the system is topologically trivial.
In order to understand the situation better, the authors compute, using a self-consistent procedure, the mass taking inter-surface (top and bottom) screened Coulomb-interactions into account. They find within their self-consistent scheme that the mass does not change change with thickness, although it does get very small with increasing thickness. They then go on to present ARPES data on the spectra of the Bi2Se3-system they consider, and conclude that a visible gap opens up for thicknesses around 6 QL and that the system is topoligically trivial.
I have several comments to this paper.
1. The mass they compute does not change sign but becomes extremely small rapidly with increasing thickness. For instance, by the time one gets to 11 QL, it would be down by a factor of roughly e^{-8.5} = 0.0002. In ARPES, one does not get the sign og m, only the abolute value. ARPES is not suited for looking for sign changes in m, and can therefore not rule out sign changes either.  The black lines are the EDCs of the ARPES spectra, while the orange lines and red lines are the Lorentzian fitting of the single peak and the total EDCs. The FWHM of fitting results is also considered as an additional error in the error bar in Fig. 3.

Response to Reviewer #2:
Overall comments: 'This paper addresses, both theoretically and experimentally (ARPES) a very timely issue, namely the role of many-body effects on the spectra of purported topological insulators. 'In several previous works, the effects on the mass gap in the single-particle spectrum of such systems has been studied as a function of thickness of samples, i.e., as a function of quintuplet layers (QL). It has been found that this mass gap can change sign from positive (topologically trivial) to negative (topologically non-trivial) as thickness is varied. The authors of the present paper point out that i) the gap reclosing (necessary for a change of sign in m) has not been observed, ii) the experimentally reported mass gaps are far larger than predicted, and iii) the absence of ballistic 1D edge states means that the system is topologically trivial. 'In order to understand the situation better, the authors compute, using a self-consistent procedure, the mass taking inter-surface (top and bottom) screened Coulombinteractions into account. They find within their self-consistent scheme that the mass does not change with thickness, although it does get very small with increasing thickness. They then go on to present ARPES data on the spectra of the Bi2Se3-system they consider, and conclude that a visible gap opens up for thicknesses around 6 QL and that the system is topologically trivial. 'I have several comments to this paper.' Authors' response: We greatly thank the reviewer for her/his positive overall comments on our work. We made some improvement in revised manuscript. The supplemental information with the details of theoretical derivation, analysis of error and the energy band fitting is also added. Below we addressed all the concerns raised by the reviewer.

Comment 2.1:
'The mass they compute does not change sign but becomes extremely small rapidly with increasing thickness. For instance, by the time one gets to 11 QL, it would be down by a factor of roughly e^{-8.5} = 0.0002. In ARPES, one does not get the sign of m, only the absolute value. ARPES is not suited for looking for sign changes in m, and can therefore not rule out sign changes either.'

Authors' response:
does not change its sign with our theoretical and experimental results. It is true that ARPES is not a suitable method to confirm the sign of in the Dirac system since the energy band ± = ± ℏ + does depends on the absolute value, rather than the sign of . We rule out the sign change based on the following calculation and experimental results: 1. The error analysis (see our response of the next comment) confirms that the sign of will not change in the case of our mass theory.
2. If the sign of changes, the absolute value of will evolute non-monotonically, which can be detected by our ARPES. For example, the (Bi1-xInx)2Se3 with thickness of larger than 6 QL show a non-monotonic evolution with In-doping as shown in Fig. R4a [Nano Lett. 19, 4627 (2019).]. However, this non-monotonic evolution does not appear in our ARPES results and previous work [Nat. Phys. 6, 584-588 (2010)]. To further confirm this result, we extracted the energy gap in the Bi2Se3 films with the thickness of 1.5 and 2.5 QL in reviewed manuscript, and the phenomenon of gap reclosing at 2.5 QL predicted by previous theory is not observed as shown in Fig. R4b (Fig. 3b) All listed experimental results demonstrate that the single-particle band theory is not sufficient to describe the band structure and transport properties in ultrathin Bi2Se3 TI films. Therefore, we suspect that the mass gap may come from the influence of the Coulomb interaction. Within the framework of our theory, the energy gap originates from the breaking of chiral symmetry instead of the time-inversion symmetry for one band of a Kramer pair, which is similar to the case in graphene with electron-electron interaction and the Kekulé-Ordered graphene [Phys. Rev. Lett. 126, 206804 (2021)].
For the 5 QL Bi2Se3 film, the measured mass gap is about 40 meV. The mass gaps for the samples with thickness above 6 QL are much smaller. Such small gap can be easily disturbed by the scattering and temperature. Even in the case of quantum anomalous Hall effect [Science 329, 659 (2010)., Science 340, 167 (2013).], the magnetic-doping induced gap is about 50 meV, which needs the low temperature of 30 mK for the transport measurements. Therefore, the magnitude of the energy gaps for the samples with thickness of above 6 QL cannot be detected, and will be not discussed here.

Comment 2.2:
'Their Eq. 10 is fitted to data in Fig. 3. Given the smallness of the mass, how confident should one be in the estimate, and the sign, given the many approximations involved both in the way they do their self-consistent calculation, as well as to factors ignored by the self-consistent approach?' Authors' response: We really thank the referee for this illuminating comment, which pushed us to find a new and more accurate approximation in the revised manuscript. All used approximations and the details of calculations are shown in the revised supplemental information. The relative errors of each approximation are also discussed, and the largest one is less than 3.5% (about 10 meV at the thickness of 2QL and even smaller at others, which is smaller than the energy resolution of our ARPES). All approximations we made will only affect the magnitude of the final results, and the sign of m does not change. Here we list all methods of approximations with relative error.
1. We ignored the wave-function renormalization functions , in Eq. (S1.3), and the mass term in Eq. (S2.5) in treated as a real number , ≈ 0,0 = . This approximation reduced the accuracy of the self-energy when ≠ 0, but has much less impact on the problem of the interactioninduced mass gap at the Gamma point and is widely used [references 1-5 in supplemental information].
2. The Coulomb interaction in our system is hard to calculate, and we find an expression as Eq. (S1.1) which deviates very little (relative error <10 -5 ) from the numerical result (see Fig. S1).
3. Zero-temperature approximation is used to simplify the polarization function in Eq. (S2.13). Our experiments were conducted at temperature of 5 K, which correspond to a 4 meV broadening of Fermi-Dirac distribution from 1% to 99% at the Fermi level. This broad is much smaller than all energy scales that we discussed, and therefore can be ignored. We also notice that the ARPES spectra were measured at room temperature in previous work [Nat. Phys. 6, 584-588 (2010)] and their fitting results of the mass gap are quite close to ours, which implies that the mass gap will not sensitively change with temperature.
4. The Lorentzian peak is approximated as a delta-function in the integral of in Eq. (S1.4). Compared with previous manuscript, this approximation has been improved. To ensure the accuracy of this approximation, the full-width at half maximum (FWHM) of this Lorentzian peak should as little as possible. In previous manuscript, we treat the term in Eq. (S1.3) as with FWHM= √2 . We use the new method in the revised manuscript to treat the term as The latter one has a narrower linewidth. The fluctuation of near = 0 in Eq. (S1.4) originates from the Green function Therefore, the relative error of delta-function approximation is less than ℏ ≈ 3.5%.
Meanwhile, in the approximation methods above, all the omitted parts do not contain the opposite sign, which makes the final result will not change the sign of .

Comment 2.3:
'It is rare to see papers in such high-profile journals as Nature written in such poor English. The manuscript is rife with basic grammatical errors and poorly formulated sentences throughout. The authors need to remedy this before resubmitting it to any scientific journal using the English language.'

LIST OF CHANGES
All changes are highlighted in the manuscript, and we list them here one by one.
1. The supplemental information with the details of the derivation of our theory and the peak fitting of ARPES spectra is added in the additional file.
2. Line 1 (In the reviewed manuscript, same below): 'Bi2Se3 thin films' is changed to 'Bi2Se3 ultrathin films' in the title.  to 'In previous works on Bi2Se3 ultrathin films 11, 12 , … It seems that the single-particle band theory is not sufficient to describe the band structure and transport properties in ultrathin Bi2Se3 40. Line 123: The off-diagonal part is added, and the = − (1 − 2 ) + 2 is changed to 41. Line 126: < < is changed to − < < . 59. Line 158-161: 'The energy gap opened below 6 QL consistent with previous work 13 , and the phenomenon of energy gap reclosing described in the previous theory is not found 11 .' is changed to 'To confirm whether the gap will reclose or not at about 2.5 QL predicted by the previous theory 11 , …, the non-monotonic evolution of the energy gap around the thickness of 2-3 QL is not found.' 60. Line 162: 'Obviously,' is changed to 'We can clearly see that'.
61. Line 163-165: 'However, it has been shown that the band bending effect at the interface between Bi2Se3 and vacuum is less than 0.1 eV when the thickness is less than 7 QL 25 . The band bending effect is much smaller at the interface between Bi2Se3 and vacuum.' is changed to 'However, …, and it should be much smaller at the interface between Bi2Se3 and silicon.' 62. Line 165-167: 'the band shifting cannot be explained by the band bending effect because it is larger than 0.1 eV as long as the thickness is thinner than 6 QL.' is changed to 'As shown in Fig. 3a  effect-induced one. In the case of Bi2Se3, the interaction-induced gap decays exponentially with the decay length of /2 ≈ 1.5 nm (from Eq. (10) in main text), while the decay length of size-effect induced gap is about 50 nm [Eq. (12) in Phys. Rev. Lett. 101, 246807 (2008)]. This difference makes the expected region may appear in thicker samples. Another way to achieve this region is to adjust the ratio of Se vacancy to reduce the in Eq. 10.